Papers with sample efficiency of pretrained masked language models
The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the rise of the prompting paradigm with the scaling breakthrough of very large language models, understanding the mechanism of model fine-tuning remains an important endeavor. |
| Approach: | They analyze the masked language modeling pretraining objective function from the perspective of the Distributional Hypothesis and examine whether the distributional property leads to better sample efficiency and better generalization capability of pretrained models. |
| Outcome: | The proposed model pretraining objective function improves sample efficiency and generalization capability but does not explain the generalization ability of natural language models. |